On using predictive-ability tests in the selection of time-series prediction models: A Monte Carlo evaluation

To select a forecast model among competing models, researchers often use ex-ante prediction experiments over training samples. Following Diebold and Mariano (1995), forecasters routinely evaluate the relative performance of models with accuracy tests may base their selection on test significance top comparing errors. With extensive Monte Carlo analysis, we investigated whether this practice favors simpler more complex ones, without gains in accuracy. We simulated autoregressive moving-average model, self-exciting threshold vector autoregression. considered two variants Diebold–Mariano test, by Giacomini White (2006), F-test Clark McCracken (2001), Akaike information criterion, pure training-sample evaluation. The findings showed some for small samples when applying tests, particularly Clark–McCracken bootstrapped tests. Evidence against testing procedure dominated, however, evaluations performed best many cases.